{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor Types in TensorFlow\n",
    "\n",
    "(You can also read this article on our website, [Easy-TensorFlow](http://www.easy-tensorflow.com/basics/tensor-types))\n",
    "\n",
    "In the previous post, we read about the concepts of __Graph__ and __Session__ which describes the way the data flows in TensorFlow. One of the first questions you might have while learning a new framework is of any new data structure that should used. TensorFlow does have its own data structure for the purpose of performance and ease of use. Tensor is the data structure used in Tensorflow (remember TensorFlow is the flow of tensors in a computational graph) and it is at the core of TensorFlow. TensorFlow programs use a tensor data structure to represent all data — only tensors are passed between operations in the computation graph. You can think of a TensorFlow tensor as an n-dimensional array or list.\n",
    "\n",
    "In this tutorial, we'll take a look at some of the __Tensor Types__ used in TensorFlow. The speciall ones commonly used in creating neural network models are namely ___Constant___, ___Variable___, and ___Placeholder___. \n",
    "\n",
    "This will also help us to shed some light on some of the points and questions left unanswered in the previous post.\n",
    "\n",
    "Remember that we need to import the TensorFlow library at the very beginning of our code using the line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "## 1. Constant \n",
    "\n",
    "As the name speaks for itself, __Constants__ are used as constants. They create a node that takes value and it does not change. You can simply create a constant tensor using __tf.constant__. It accepts the five arguments:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.constant(value, dtype=None, shape=None, name='Const', verify_shape=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's take a look at a very simple example.\n",
    "\n",
    "### Example 1:\n",
    "Let's create two constants and add them together. Constant tensors can simply be defined with a value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "# create graph\n",
    "a = tf.constant(2)\n",
    "b = tf.constant(3)\n",
    "c = a + b\n",
    "# launch the graph in a session\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect! Now Let's look at the created graph and generated data types:\n",
    "<img src=\"files/files/2_1.png\" width=\"1000\" height=\"2000\" >\n",
    "___Fig1. ___ __Left:__ generated graph visualized in Tensorboard, __Right:__ generated variables (screenshot captured from PyCharm debugger when running in debug mode)\n",
    "\n",
    "In the figure, we created 3 tensors with __\"Python-names\"__ _a_, _b_, and _c_. As we didn't define any __\"TensorFlow-name\"__ for them, TensorFlow assigns some default names to them which are observed in the graph: __const__ and __const_1__ for the input constants and __add__ for the output of the addition operation. We can easily modify it and define custom names as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "# create graph\n",
    "a = tf.constant(2, name='A')\n",
    "b = tf.constant(3, name='B')\n",
    "c = tf.add(a, b, name='Sum')\n",
    "# launch the graph in a session\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time the graph is created with the required tensor names:\n",
    "<img src=\"files/files/2_2.png\" width=\"1000\" height=\"2000\" >\n",
    "___Fig2. ___ generated graph (Left) and variables (Right) with the modified names\n",
    "\n",
    "\n",
    "Constants can also be defined with different types (integer, float, etc.) and shapes (vectors, matrices, etc.). The next example has one constant with type 32bit float and another constant with shape 2X2.\n",
    "\n",
    "\n",
    "### Example 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.3\n",
      "[[1 2]\n",
      " [3 4]]\n"
     ]
    }
   ],
   "source": [
    "s = tf.constant(2.3, name='scalar', dtype=tf.float32)\n",
    "m = tf.constant([[1, 2], [3, 4]], name='matrix')\n",
    "# launch the graph in a session\n",
    "with tf.Session() as sess:\n",
    "    print(sess.run(s))\n",
    "    print(sess.run(m))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Variable\n",
    "\n",
    "Variables are stateful nodes which output their current value; meaning that they can retain their value over multiple executions of a graph. They have a number of useful features such as:\n",
    "\n",
    "- They can be __saved__ to your disk during and after training. This allows people from different companies and groups to collaborate as they can save, restore and send over their model parameters to other people. (read more on [Save and Restore Tutorial](https://github.com/easy-tensorflow/easy-tensorflow/blob/master/1_TensorFlow_Basics/Tutorials/4_Save_and_Restore.ipynb))\n",
    "- By default, gradient updates (used in all neural networks) will apply to all variables in your graph. In fact, variables are the things that you want to tune in order to minimize the loss. \n",
    "\n",
    "These two features make variables suitable to be used as the network parameters (i.e. weights and biases). You might ask, what are the differences between variables and constants? Well there are two major differences:\n",
    "\n",
    "1. Constants are (guess what!), constants. As their name states, their value doesn't change. We'd usually need our network parameters to be updated and that's where the __variable__ comes into play.\n",
    "\n",
    "2. Constants are stored in the graph definition which makes them memory-expensive. In other words, constants with millions of entries makes the graph slower and resource intensive.\n",
    "\n",
    "\n",
    "Again, it's important to remember that creating a variables is an operation (look at the Fig. 2 of the first tutorial for a quick recap). We execute these operations in the session and get the output value of the operations.\n",
    "\n",
    "### 2.1. Create Variables\n",
    "\n",
    "To create a variable, we should use __tf.Variable__ as:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a variable.\n",
    "w = tf.Variable(<initial-value>, name=<optional-name>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some examples of creating scalar and matrix variables are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s = tf.Variable(2, name=\"scalar\") \n",
    "m = tf.Variable([[1, 2], [3, 4]], name=\"matrix\") \n",
    "W = tf.Variable(tf.zeros([784,10]))"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Variable __W__ defined above creates a matrix with 784 rows and 10 columns which will be initialized with zeros. This can be used as a weight matrix of a feed-forward neural network (or even in a linear regression model) from a layer with 784 neuron to a layer with 10 neuron. We'll see more of this later in this turorial.\n",
    "\n",
    "__*Note:__ We use tf.Variable() with uppercase \"V\", and tf.constant with lowercase \"c\". You don't necessarily need to know the reason, but it's simply because tf.constant is an op, while tf.Variable is a class with multiple ops.\n",
    "\n",
    "__* IMPORTANT Note:__ Calling tf.Variable to create a variable is the old way of creating a variable. TensorFlow recommends to use the wraper __tf.get_variable__ instead as it accepts the name, shape, etc as its arguments with many more as follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.get_variable(name,\n",
    "                shape=None,\n",
    "                dtype=None,\n",
    "                initializer=None,\n",
    "                regularizer=None,\n",
    "                trainable=True,\n",
    "                collections=None,\n",
    "                caching_device=None,\n",
    "                partitioner=None,\n",
    "                validate_shape=True,\n",
    "                use_resource=None,\n",
    "                custom_getter=None,\n",
    "                constraint=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some examples are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s = tf.get_variable(\"scalar\", initializer=tf.constant(2)) \n",
    "m = tf.get_variable(\"matrix\", initializer=tf.constant([[0, 1], [2, 3]]))\n",
    "W = tf.get_variable(\"weight_matrix\", shape=(784, 10), initializer=tf.zeros_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Initialize Variables\n",
    "\n",
    "Just like most programming languages, Variables need to be initialized before being used. TensorFlow, while not being a language, is no exception to this rule. To initialize variables, we have to invoke a __variable initializer operation__ and run the operation on the session. This is the easiest way to initialize variables all variables at once.\n",
    "\n",
    "The following toy example shows how we can add an op to initialize the variables.\n",
    "\n",
    "### Example 3:\n",
    "\n",
    "Create two variables and add them together. Then print out their values and the summation result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = tf.get_variable(name=\"var_1\", initializer=tf.constant(2))\n",
    "b = tf.get_variable(name=\"var_2\", initializer=tf.constant(3))\n",
    "c = tf.add(a, b, name=\"Add1\")\n",
    "\n",
    "# launch the graph in a session\n",
    "with tf.Session() as sess:\n",
    "    # now let's evaluate their value\n",
    "    print(sess.run(a))\n",
    "    print(sess.run(b))\n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__FailedPreconditionError: Attempting to use uninitialized value__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon executing the program, we run into __FailedPreconditionError: Attempting to use uninitialized value__. This is because we tried to evaluate the variables before initializing them. Let's correct the code by first initializing all the variables and then proceed to evaluate them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "3\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "# create graph\n",
    "a = tf.get_variable(name=\"A\", initializer=tf.constant(2))\n",
    "b = tf.get_variable(name=\"B\", initializer=tf.constant(3))\n",
    "c = tf.add(a, b, name=\"Add\")\n",
    "# add an Op to initialize global variables\n",
    "init_op = tf.global_variables_initializer()\n",
    "\n",
    "# launch the graph in a session\n",
    "with tf.Session() as sess:\n",
    "    # run the variable initializer operation\n",
    "    sess.run(init_op)\n",
    "    # now let's evaluate their value\n",
    "    print(sess.run(a))\n",
    "    print(sess.run(b))\n",
    "    print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a quick look at the graph and generated variables:\n",
    "<img src=\"files/files/2_3.png\" width=\"1000\" height=\"2000\" >\n",
    "___Fig3. ___ generated graph (Left) and variables (Right)\n",
    "\n",
    "The figure above shows that two blue boxes are the generated and they represent the variables (compare them with constant nodes in Fig. 2). These two variables are added together using the \"Add\" operation.\n",
    "\n",
    "\n",
    "__*Note:__ Variables are usually used for weights and biases in neural networks.\n",
    "\n",
    "- __Weights__ are usually initialized from a normal distribution using `tf.truncated_normal_initializer()`.\n",
    "\n",
    "- __Biases__ are usually initialized from zeros using `tf.zeros_initializer()`.\n",
    "\n",
    "Let's look at a very simple example of creating weight and bias variables with proper initialization:\n",
    "\n",
    "### Example 4:\n",
    "\n",
    "Create the weight and bias matrices for a fully-connected layer with 2 neuron to another layer with 3 neuron. In this scenario, the weight and bias variables must be of size $[2, 3]$ and 3 respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "weights = [[-0.00376599 -0.00506956  0.00082394]\n",
      " [ 0.0016487   0.00981423 -0.00226094]]\n",
      "biases = [0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "# create graph\n",
    "weights = tf.get_variable(name=\"W\", shape=[2,3], initializer=tf.truncated_normal_initializer(stddev=0.01))\n",
    "biases = tf.get_variable(name=\"b\", shape=[3], initializer=tf.zeros_initializer())\n",
    "\n",
    "# add an Op to initialize global variables\n",
    "init_op = tf.global_variables_initializer()\n",
    "\n",
    "# launch the graph in a session\n",
    "with tf.Session() as sess:\n",
    "    # run the variable initializer\n",
    "    sess.run(init_op)\n",
    "    # now we can run our operations\n",
    "    W, b = sess.run([weights, biases])\n",
    "    print('weights = {}'.format(W))\n",
    "    print('biases = {}'.format(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Placeholder:\n",
    "\n",
    "Placeholders are more basic than a variable. It is simply a variable that we asign data in a future time. Placeholders are nodes whose value is fed in at execution time. If we have inputs to our network that depend on some external data and we don't want our graph to depend on any real value while developing the graph, placeholders are the datatype we need. In fact, we can build the graph without any data. Therefore, placeholders don't need any initial value; only a datatype (such as float32) and a tensor shape so the graph still knows what to compute with even though it doesn't have any stored values yet.\n",
    "\n",
    "Some examples of creating placeholders are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = tf.placeholder(tf.float32, shape=[5])\n",
    "b = tf.placeholder(dtype=tf.float32, shape=None, name=None)\n",
    "X = tf.placeholder(tf.float32, shape=[None, 784], name='input')\n",
    "Y = tf.placeholder(tf.float32, shape=[None, 10], name='label') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run a simple example.\n",
    "\n",
    "### Example 5:\n",
    "\n",
    "Create a constant vector and a placeholder and add them together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = tf.constant([5, 5, 5], tf.float32, name='A')\n",
    "b = tf.placeholder(tf.float32, shape=[3], name='B')\n",
    "c = tf.add(a, b, name=\"Add\")\n",
    "\n",
    "with tf.Session() as sess:\n",
    "      print(sess.run(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__InvalidArgumentError: You must feed a value for placeholder tensor 'B' with dtype float and shape  $[3]$ error__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Executing the above code will run into an error. You might have guessed it, its simply because the placeholder is empty and there is no way to add an empty tensor to a constant tensor in the add operation. To solve this, we need to feed an input value to the tensor \"b\". It can be done by creating a dictionary (\"d\" in the following code) whose key(s) are the placeholders and their values are the desired value to be passed to the placeholder(s), and feeding it to an argument called \"feed_dict\". In our example, say we want to pass $[1, 2, 3]$ to the placeholder; the code needs to be modified as:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6. 7. 8.]\n"
     ]
    }
   ],
   "source": [
    "a = tf.constant([5, 5, 5], tf.float32, name='A')\n",
    "b = tf.placeholder(tf.float32, shape=[3], name='B')\n",
    "c = tf.add(a, b, name=\"Add\")\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # create a dictionary:\n",
    "    d = {b: [1, 2, 3]}\n",
    "    # feed it to the placeholder\n",
    "    print(sess.run(c, feed_dict=d)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generated graph and variables are as follows:\n",
    "<img src=\"files/files/2_4.png\" width=\"700\" height=\"1400\" >\n",
    "___Fig4. ___ generated graph (Left) and variables (Right)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "So far so good? To make it more interesting and challenging lets get our hands dirty!\n",
    "\n",
    "\n",
    "## Creating a Neural Network\n",
    "\n",
    "Now, we have all the required materials to start building a toy feed-forward neural network with one hidden layer and 200 hidden units (neurons). The computational graph in Tensorflow will be:\n",
    "\n",
    "<img src=\"files/files/2_5.png\" width=\"300\" height=\"600\" >\n",
    "___Fig5. ___ Schematic of the graph for one layer of the neural network\n",
    "\n",
    "How many operations (or nodes) do you see in this graph? Six! right? The three circles (X, W, b) and three rectangles. We'll go through each of them and will discuss the best way to implement it.\n",
    "\n",
    "Let's start with the input, X. This can be an input of any type, such as images, signals, etc. The general approach is to feed all inputs to the network and train the trainable parameters (here, W and b) by backpropagating the error signal. Ideally, you need to feed all inputs together, compute the error, and update the parameters. This process is called \"Gradient Descent\".\n",
    "\n",
    "*Side Note: In real-world problems, we have thousands and millions of inputs which makes gradient descent computationally expensive. That's why we split the input set into several shorter pieces (called mini-batch) of size B (called mini-batch size) inputs, and feed them one by one. This is called \"Stochastic Gradient Descent\". The process of feeding each mini-batch of size B to the network, back-propagating errors, and updating the parameters (weights and biases) is called an iteration.\n",
    "\n",
    "We generally use Placeholders for inputs so that we can build the graph without any real value in context. The only point is that you need to choose the proper size for the input. Here, we have a feed-forward neural network, and let's assume inputs of size 784 (similar to 28x28 images of MNIST data). The input placeholder can be written as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create the input placeholder\n",
    "X = tf.placeholder(tf.float32, shape=[None, 784], name=\"X\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might wonder why is the shape=$[None, 784]$?!\n",
    "\n",
    "Well, that's the tricky part! Please read the above side note again. We need to feed B images of size 784 to the network in each training iteration as one batch. So the placeholder needs to be of shape=$[B, 784]$. Defining the placeholder shape as $[None, 784]$ means that we can feed any number of images of size 784 (not B images necessarily). This is especially helpful in the evaluation time where we need to feed all validation or test images to the network and compute the performance on all of them.\n",
    "\n",
    "Enough with the placeholders. Let's move on to the network parameters, W, and b. As explained in the Variable section above, they have to be defined as variables. Since in Tensorflow, gradient updates will be applied to the graph variables, by default. As mentioned, variables need to be initialized.\n",
    "\n",
    "*Note: Generally, weights (W) are initialized randomly, in it's the simplest form from a normal distribution, say normal distribution with zero mean and standard deviation of 0.01. Biases (b) can be initialized as small constant values, such as 0.\n",
    "\n",
    "Since the input dimension is 784 and we have 200 hidden units, the weight matrix will be of size $[784, 200]$. We also need 200 biases, one for each hidden unit. The code will be like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create weight matrix initialized randomely from N(0, 0.01)\n",
    "weight_initer = tf.truncated_normal_initializer(mean=0.0, stddev=0.01)\n",
    "W = tf.get_variable(name=\"Weight\", dtype=tf.float32, shape=[784, 200], initializer=weight_initer)\n",
    "\n",
    "# create bias vector of size 200, all initialized as zero\n",
    "bias_initer =tf.constant(0., shape=[200], dtype=tf.float32)\n",
    "b = tf.get_variable(name=\"Bias\", dtype=tf.float32, initializer=bias_initer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's move on to the rectangle operations. We must multiply input X$_{[None, 784]}$ and weight matrix W$_{[784, 200]}$ which gives a tensor of size $[None, 200]$, then add the bias vector b$_{[200]}$ and eventually pass the final tensor from a ReLU non-linearity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create MatMul node\n",
    "x_w = tf.matmul(X, W, name=\"MatMul\")\n",
    "# create Add node\n",
    "x_w_b = tf.add(x_w, b, name=\"Add\")\n",
    "# create ReLU node\n",
    "h = tf.nn.relu(x_w_b, name=\"ReLU\")\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, we are all set. The created graph looks like this:\n",
    "<img src=\"files/files/2_6.png\" width=\"400\" height=\"800\" >\n",
    "___Fig6. ___ Data flow graph of the neural network created in Tensorflow\n",
    "\n",
    "But how can we visualize this graph? How do you create this figure? That's the magic of __Tensorboard__. It's thoroughly explained in our next article.\n",
    "\n",
    "Before closing it, let's run a session on this graph (using 100 images generated by random pixel values) and get the output of hidden units (h). Below is the complete code:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import the tensorflow library\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "# create the input placeholder\n",
    "X = tf.placeholder(tf.float32, shape=[None, 784], name=\"X\")\n",
    "weight_initer = tf.truncated_normal_initializer(mean=0.0, stddev=0.01)\n",
    "\n",
    "# create network parameters\n",
    "W = tf.get_variable(name=\"Weight\", dtype=tf.float32, shape=[784, 200], initializer=weight_initer)\n",
    "bias_initer =tf.constant(0., shape=[200], dtype=tf.float32)\n",
    "b = tf.get_variable(name=\"Bias\", dtype=tf.float32, initializer=bias_initer)\n",
    "\n",
    "# create MatMul node\n",
    "x_w = tf.matmul(X, W, name=\"MatMul\")\n",
    "# create Add node\n",
    "x_w_b = tf.add(x_w, b, name=\"Add\")\n",
    "# create ReLU node\n",
    "h = tf.nn.relu(x_w_b, name=\"ReLU\") \n",
    "\n",
    "# Add an Op to initialize variables\n",
    "init_op = tf.global_variables_initializer()\n",
    "\n",
    "# launch the graph in a session\n",
    "with tf.Session() as sess:\n",
    "    # initialize variables\n",
    "    sess.run(init_op)\n",
    "    # create the dictionary:\n",
    "    d = {X: np.random.rand(100, 784)}\n",
    "    # feed it to placeholder a via the dict \n",
    "    print(sess.run(h, feed_dict=d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running this code will print out h$_{[100, 200]}$ which are the outputs of 200 hidden units in response to 100 images; i.e. 200 features extracted from 100 images.\n",
    "\n",
    "We'll continue constructing the loss function and creating the optimizer operations in the next articles. However, we need to learn Tensorboard first to use its amazing features in our neural network code.\n",
    "\n",
    "I hope this post has helped you to understand how to use different __Tensor Types__ in TensorFlow. Thank you so much for reading! If you have any questions, feel free to leave a comment in our [webpage](http://www.easy-tensorflow.com/basics/tensor-types). You can also send us feedback through the [__contacts__](http://www.easy-tensorflow.com/contacts) page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
